Record workflow version control information post-install #4142

MetRonnie · 2021-03-24T17:03:01Z

These changes close #3849

On installation of a workflow, will record the version control status to log/version/vcs.conf and any uncommitted diff to log/version/uncommitted.diff.

Supports git and svn.

Requirements check-list

I have read CONTRIBUTING.md and added my name as a Code Contributor.
Contains logically grouped changes (else tidy your branch by rebase).
Does not contain off-topic changes (use other PRs for other changes).
Appropriate tests are included (unit and/or functional).
Appropriate change log entry included
(master branch) I have opened a documentation PR at Autodocument post-install plugins cylc-doc#236
No dependency changes.

MetRonnie · 2021-03-25T13:14:07Z

cylc/flow/post_install/log_vc_info.py

+    """
+    if not info:
+        raise ValueError("Nothing to write")
+    info_file = Path(run_dir, 'log', 'version', 'vcs.conf')


I can't see a way to get the timestamp of the installation, perhaps this needs to be added to the entry point arguments?

datamel

Minor comments. I have tested the change as working, read the code and run the tests. Thanks Ronnie.

datamel · 2021-03-30T08:41:01Z

cylc/flow/post_install/log_vc_info.py

+    return None
+
+
+def _run_cmd(vcs: str, args: Iterable[str], cwd: Union[Path, str]) -> str:


Worth making use of run_cmd in remote.py? Could adapt it if you need to pass the cwd?

Ideally, but it wouldn't quite work here due to the difference in error handling

cylc/flow/post_install/log_vc_info.py

MetRonnie · 2021-03-30T10:05:31Z

Just remembered a question I had: do we want to save the diff of untracked files for a git source dir? This is actually not the default behaviour of git, git diff HEAD only gets the diff of tracked files (unstaged + staged)

datamel · 2021-03-30T12:51:22Z

Just remembered a question I had: do we want to save the diff of untracked files for a git source dir? This is actually not the default behaviour of git, git diff HEAD only gets the diff of tracked files (unstaged + staged)

Could those untracked files make a difference to the running of the workflow? For example, if they had configured their .cylcignore file to ignore directories for the installation but this was not under source control, they may wish to know about it. Although perhaps this could lead to the log filling with unwanted diffs... difficult choice. I guess by marking files as untracked they should assume it would not be included in the diff, perhaps a one line warning at the top reminding them that there may be untracked changed?

MetRonnie · 2021-03-30T14:18:45Z

Could those untracked files make a difference to the running of the workflow?

Yes

I guess by marking files as untracked they should assume it would not be included in the diff, perhaps a one line warning at the top reminding them that there may be untracked changed?

I don't think there is a way of marking files as untracked. It's pretty much only newly-created files that haven't been git added yet that are untracked.

The untracked files will still show up in the vcs.conf file with ?? status, e.g.

status = """
?? foo/bar.txt
"""

cylc/flow/post_install/__init__.py

oliver-sanders

Looking good, just need to give it a spin.

oliver-sanders · 2021-04-22T09:12:09Z

cylc/flow/post_install/__init__.py

+Built In Plugins
+----------------
+
+Cylc Flow provides the following post-install plugins:


We can document the pre-install and post-install entry points together as "install plugins" since they have related functionality and plugins may use both (e.g. cylc-rose).

tests/unit/post_install/test_log_vc_info.py

oliver-sanders

Tested with Git, all good.

Needs a quick changelog entry.
Can you squash down commits, namely "Tidy" ones.

oliver-sanders · 2021-04-26T09:38:41Z

cylc/flow/install_plugins/log_vc_info.py

+
+"""Record version control information on workflow install.
+
+The supported version control systems are git and svn.


From cylc/cylc-doc#236:

Could do with a little more documentation here, a quick outline of:

Where the version control information is stored.

The format of this file.

The meaning of each field.

Can be quite short.

Added: https://github.com/cylc/cylc-flow/pull/4142/files#diff-050731cd5a466ce73267f816ff7c03dc72ad30e21d92d29db741db0395ba64fc

Plus tidy

oliver-sanders · 2021-04-27T11:51:39Z

cylc/flow/install_plugins/log_vc_info.py

+   repository version = "2.8.0-dirty"
+   commit = "e5dc6573dd70cabd8f973d1535c17c29c026d553"
+   working copy root path = "~/cylc-src/my-workflow-git"
+   status = \"\"\"


I don't think these quotes should be escaped here?

Because the whole docstring is a triple quote string? Rendering fine:

MetRonnie added this to the cylc-8.0b1 milestone Mar 24, 2021

MetRonnie self-assigned this Mar 24, 2021

This comment has been minimized.

Sign in to view

MetRonnie force-pushed the record-vc branch 4 times, most recently from 670f005 to 4488a2e Compare March 24, 2021 18:37

MetRonnie marked this pull request as draft March 24, 2021 18:37

MetRonnie force-pushed the record-vc branch 4 times, most recently from cbd0350 to 7979ff8 Compare March 25, 2021 13:12

MetRonnie commented Mar 25, 2021

View reviewed changes

MetRonnie marked this pull request as ready for review March 25, 2021 13:15

MetRonnie requested review from datamel and oliver-sanders March 25, 2021 13:15

MetRonnie force-pushed the record-vc branch from 7979ff8 to 789b026 Compare March 25, 2021 16:16

MetRonnie marked this pull request as draft March 26, 2021 16:58

This comment has been minimized.

Sign in to view

MetRonnie marked this pull request as ready for review March 29, 2021 08:25

MetRonnie force-pushed the record-vc branch from 789b026 to 57773a2 Compare March 29, 2021 08:29

datamel approved these changes Mar 30, 2021

View reviewed changes

MetRonnie mentioned this pull request Apr 6, 2021

Autodocument post-install plugins cylc/cylc-doc#236

Merged

oliver-sanders reviewed Apr 6, 2021

View reviewed changes

cylc/flow/post_install/__init__.py Outdated Show resolved Hide resolved

hjoliver modified the milestones: cylc-8.0b1, cylc-8.0b2 Apr 15, 2021

oliver-sanders reviewed Apr 22, 2021

View reviewed changes

oliver-sanders approved these changes Apr 26, 2021

View reviewed changes

oliver-sanders reviewed Apr 26, 2021

View reviewed changes

MetRonnie added 12 commits April 26, 2021 11:31

Create post_install plugin for recording workflow version control info

110e51a

Add unit tests for log_vc_info git functions

38cbb0b

Handle case where git or svn not installed

a076742

Add unit tests for log_vc_info svn functions

30e4886

Plus tidy

log_vc_info: Handle case of no base commit in git repo

ca04846

Add functional test for log_vc_info

eb82c32

Update log_vc_info to new post-install entry point args

7431bfb

Plus tidy

Fix test failure on GH Actions

cadf662

Update changelog

f622a99

Tidy

fe04ced

Group pre-configure & post-install plugins

9488be7

Address code review

7153a32

MetRonnie force-pushed the record-vc branch from 1ab6d94 to 7153a32 Compare April 26, 2021 12:16

Merge branch 'master' into record-vc

21a05de

oliver-sanders reviewed Apr 27, 2021

View reviewed changes

oliver-sanders merged commit 141e944 into cylc:master Apr 28, 2021

MetRonnie deleted the record-vc branch April 28, 2021 15:33

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Record workflow version control information post-install #4142

Record workflow version control information post-install #4142

MetRonnie commented Mar 24, 2021 •

edited

Loading

This comment has been minimized.

MetRonnie Mar 25, 2021

This comment has been minimized.

datamel left a comment

datamel Mar 30, 2021

MetRonnie Mar 30, 2021

MetRonnie commented Mar 30, 2021

datamel commented Mar 30, 2021

MetRonnie commented Mar 30, 2021 •

edited

Loading

oliver-sanders left a comment

oliver-sanders Apr 22, 2021

oliver-sanders left a comment

oliver-sanders Apr 26, 2021

MetRonnie Apr 27, 2021

oliver-sanders Apr 27, 2021

MetRonnie Apr 27, 2021

		return None


		def _run_cmd(vcs: str, args: Iterable[str], cwd: Union[Path, str]) -> str:


		"""Record version control information on workflow install.

		The supported version control systems are git and svn.

Record workflow version control information post-install #4142

Record workflow version control information post-install #4142

Conversation

MetRonnie commented Mar 24, 2021 • edited Loading

This comment has been minimized.

Choose a reason for hiding this comment

This comment has been minimized.

datamel left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MetRonnie commented Mar 30, 2021

datamel commented Mar 30, 2021

MetRonnie commented Mar 30, 2021 • edited Loading

oliver-sanders left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

oliver-sanders left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MetRonnie commented Mar 24, 2021 •

edited

Loading

MetRonnie commented Mar 30, 2021 •

edited

Loading